Robust nonparametric regression based on deep ReLU neural networks

In this paper, we consider robust nonparametric regression using deep neural networks with ReLU activation function. While several existing theoretically justified methods are geared towards robustness against identical heavy-tailed noise distributions, the rise of adversarial attacks has emphasized the importance of safeguarding estimation procedures against systematic contamination. We approach this statistical issue by shifting our focus towards estimating conditional distributions. To address it robustly, we introduce a novel estimation procedure based on $\ell$-estimation. Under a mild model assumption, we establish general non-asymptotic risk bounds for the resulting estimators, showcasing their robustness against contamination, outliers, and model misspecification. We then delve into the application of our approach using deep ReLU neural networks. When the model is well-specified and the regression function belongs to an $\alpha$-H\"older class, employing $\ell$-type estimation on suitable networks enables the resulting estimators to achieve the minimax optimal rate of convergence. Additionally, we demonstrate that deep $\ell$-type estimators can circumvent the curse of dimensionality by assuming the regression function closely resembles the composition of several H\"older functions. To attain this, new deep fully-connected ReLU neural networks have been designed to approximate this composition class. This approximation result can be of independent interest.Comment: 40 page

Estimator selection for regression functions in exponential families with application to changepoint detection

We observe $n$ independent pairs of random variables $(W_{i}, Y_{i})$ for which the conditional distribution of $Y_{i}$ given $W_{i}=w_{i}$ belongs to a one-parameter exponential family with parameter ${\mathbf{\gamma}}^{*}(w_{i})\in{\mathbb{R}}$ and our aim is to estimate the regression function ${\mathbf{\gamma}}^{*}$. Our estimation strategy is as follows. We start with an arbitrary collection of piecewise constant candidate estimators based on our observations and by means of the same observations, we select an estimator among the collection. Our approach is agnostic to the dependencies of the candidate estimators with respect to the data and can therefore be unknown. From this point of view, our procedure contrasts with other alternative selection methods based on data splitting, cross validation, hold-out etc. To illustrate its theoretical performance, we establish a non-asymptotic risk bound for the selected estimator. We then explain how to apply our procedure to the changepoint detection problem in exponential families. The practical performance of the proposed algorithm is illustrated by a comparative simulation study under different scenarios and on two real datasets from the copy numbers of DNA and British coal disasters records

Robust estimation of a regression function in exponential families

We observe $n$ pairs of independent random variables $X_{1}=(W_{1},Y_{1}),\ldots,X_{n}=(W_{n},Y_{n})$ and assume, although this might not be true, that for each $i\in\{1,\ldots,n\}$, the conditional distribution of $Y_{i}$ given $W_{i}$ belongs to a given exponential family with real parameter $\theta_{i}^{\star}=\boldsymbol{\theta}^{\star}(W_{i})$ the value of which is an unknown function $\boldsymbol{\theta}^{\star}$ of the covariate $W_{i}$. Given a model $\boldsymbol{\overline\Theta}$ for $\boldsymbol{\theta}^{\star}$, we propose an estimator $\boldsymbol{\widehat \theta}$ with values in $\boldsymbol{\overline\Theta}$ the construction of which is independent of the distribution of the $W_{i}$. We show that $\boldsymbol{\widehat \theta}$ possesses the properties of being robust to contamination, outliers and model misspecification. We establish non-asymptotic exponential inequalities for the upper deviations of a Hellinger-type distance between the true distribution of the data and the estimated one based on $\boldsymbol{\widehat \theta}$. We deduce a uniform risk bound for $\boldsymbol{\widehat \theta}$ over the class of H\"olderian functions and we prove the optimality of this bound up to a logarithmic factor. Finally, we provide an algorithm for calculating $\boldsymbol{\widehat \theta}$ when $\boldsymbol{\theta}^{\star}$ is assumed to belong to functional classes of low or medium dimensions (in a suitable sense) and, on a simulation study, we compare the performance of $\boldsymbol{\widehat \theta}$ to that of the MLE and median-based estimators. The proof of our main result relies on an upper bound, with explicit numerical constants, on the expectation of the supremum of an empirical process over a VC-subgraph class. This bound can be of independent interest

H

This paper is concerned with the H∞ filtering for a class of networked Markovian jump systems with multiple communication delays. Due to the existence of communication constraints, the measurement signal cannot arrive at the filter completely on time, and the stochastic communication delays are considered in the filter design. Firstly, a set of stochastic variables is introduced to model the occurrence probabilities of the delays. Then based on the stochastic system approach, a sufficient condition is obtained such that the filtering error system is stable in the mean-square sense and with a prescribed H∞ disturbance attenuation level. The optimal filter gain parameters can be determined by solving a convex optimization problem. Finally, a simulation example is given to show the effectiveness of the proposed filter design method

Estimating a regression function in exponential families by model selection

Let $X_{1}=(W_{1},Y_{1}),\ldots,X_{n}=(W_{n},Y_{n})$ be $n$ pairs of independent random variables. We assume that, for each $i\in\{1,\ldots,n\}$, the conditional distribution of $Y_{i}$ given $W_{i}$ belongs to a one-parameter exponential family with parameter ${\boldsymbol{\gamma}}^{\star}(W_{i})\in{\mathbb{R}}$, or at least, is close enough to a distribution of this form. The objective of the present paper is to estimate these conditional distributions on the basis of the observation ${\boldsymbol{X}}=(X_{1},\ldots,X_{n})$ and to do so, we propose a model selection procedure together with a non-asymptotic risk bound for the resulted estimator with respect to a Hellinger-type distance. When ${\boldsymbol{\gamma}}^{\star}$ does exist, the procedure allows to obtain an estimator $\widehat{\boldsymbol{\gamma}}$ of ${\boldsymbol{\gamma}}^{\star}$ adapted to a wide range of the anisotropic Besov spaces. When ${\boldsymbol{\gamma}}^{\star}$ has a general additive or multiple index structure, we construct suitable models and show the resulted estimators by our procedure based on such models can circumvent the curse of dimensionality. Moreover, we consider model selection problems for ReLU neural networks and provide an example where estimation based on neural networks enjoys a much faster converge rate than the classical models. Finally, we apply this procedure to solve variable selection problem in exponential families. The proofs in the paper rely on bounding the VC dimensions of several collections of functions, which can be of independent interest

Effect of Silicon Addition on High-Temperature Solid Particle Erosion-Wear Behaviour of Mullite-SiC Composite Refractories Prepared by Nitriding Reactive

Solid particle erosion-wear experiments on as-prepared mullite-SiC composite refractories by nitriding reactive sintering were performed at elevated temperatures, using sharp black SiC abrasive particles at an impact speed of 50 m/s and the impact angle of 90° in the air atmosphere. The effects of silicon powder addition and erosion temperature on the erosion-wear resistance of mullite-SiC composite refractories were studied. The test results reveal that Si powders caused nitriding reaction to form β-sialon whiskers in the matrix of mullite-SiC composite refractories. The erosion-wear resistance of mullite-SiC composite refractories was improved with the increase of silicon powder addition and erosion temperature, and the minimum volume erosion rate was under the condition of 12% silicon added and a temperature of 1400°C. The major erosion-wear mechanisms of mullite-SiC composite refractories were brittle erosion at the erosion temperature from room temperature to 1000°C and then plastic deformation from 1200°C to 1400°C